{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用 Keras 做 IMDB 影评分类 - RNN\n",
    "\n",
    "本文是阅读 _Deep Learning with Python_ 的笔记，记录了使用 keras 处理文本的种种模型与技巧。\n",
    "\n",
    "### 导入数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras.datasets import imdb\n",
    "from tensorflow.keras.preprocessing import sequence\n",
    "\n",
    "max_words = 10000 # 取出现频率最高的 1w 词\n",
    "maxlen = 200 # 单个序列的最大长度\n",
    "\n",
    "(input_train, y_train), (input_test, y_test) = imdb.load_data(num_words=max_words)\n",
    "\n",
    "# pad_sequences 的功能是让所有序列都一样长，长度截断，短的补零\n",
    "X_train = sequence.pad_sequences(input_train, maxlen=maxlen)\n",
    "X_test = sequence.pad_sequences(input_test, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 构建网络\n",
    "\n",
    "此处的输入序列是评论中各个词在词典中的 index 组成的序列。该 index 通过 Embedding 层之后，就得到了词向量。词向量再输入给 RNN，最后使用一个全连接层完成分类。\n",
    "\n",
    "使用多层 RNNs 需要前面一层的 RNNs 在每个时刻都产生输出，要将 `return_sequences` 置为 True。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/6\n",
      "20000/20000 [==============================] - 27s 1ms/sample - loss: 0.4550 - acc: 0.7863 - val_loss: 0.3854 - val_acc: 0.8342\n",
      "Epoch 2/6\n",
      "20000/20000 [==============================] - 26s 1ms/sample - loss: 0.2882 - acc: 0.8830 - val_loss: 0.3339 - val_acc: 0.8676\n",
      "Epoch 3/6\n",
      "20000/20000 [==============================] - 25s 1ms/sample - loss: 0.2234 - acc: 0.9165 - val_loss: 0.2822 - val_acc: 0.8840\n",
      "Epoch 4/6\n",
      "20000/20000 [==============================] - 26s 1ms/sample - loss: 0.1714 - acc: 0.9358 - val_loss: 0.2629 - val_acc: 0.8936\n",
      "Epoch 5/6\n",
      "20000/20000 [==============================] - 25s 1ms/sample - loss: 0.1346 - acc: 0.9510 - val_loss: 0.2807 - val_acc: 0.8940\n",
      "Epoch 6/6\n",
      "20000/20000 [==============================] - 26s 1ms/sample - loss: 0.1057 - acc: 0.9637 - val_loss: 0.2898 - val_acc: 0.8826\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Embedding(max_words, 128),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.GRU(128),\n",
    "    keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "\n",
    "history = model.fit(X_train, y_train, epochs=6,\n",
    "                    batch_size=32, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 12s 460us/sample - loss: 0.3691 - acc: 0.8744\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.3691024150466919, 0.87444]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里使用了一个很随意的 RNN 模型，结果显示并不如简单的词袋模型效果好。简单的词袋模型加两层全连接也能得到 89% 的精度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用 bidirectional RNNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/6\n",
      "20000/20000 [==============================] - 27s 1ms/sample - loss: 0.4613 - acc: 0.7891 - val_loss: 0.3538 - val_acc: 0.8538\n",
      "Epoch 2/6\n",
      "20000/20000 [==============================] - 26s 1ms/sample - loss: 0.2801 - acc: 0.8904 - val_loss: 0.2594 - val_acc: 0.8982\n",
      "Epoch 3/6\n",
      "20000/20000 [==============================] - 26s 1ms/sample - loss: 0.1976 - acc: 0.9263 - val_loss: 0.2593 - val_acc: 0.8942\n",
      "Epoch 4/6\n",
      "20000/20000 [==============================] - 26s 1ms/sample - loss: 0.1424 - acc: 0.9484 - val_loss: 0.2838 - val_acc: 0.8954\n",
      "Epoch 5/6\n",
      "20000/20000 [==============================] - 26s 1ms/sample - loss: 0.1061 - acc: 0.9634 - val_loss: 0.3211 - val_acc: 0.8810\n",
      "Epoch 6/6\n",
      "20000/20000 [==============================] - 26s 1ms/sample - loss: 0.0778 - acc: 0.9740 - val_loss: 0.3416 - val_acc: 0.8846\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Embedding(max_words, 128),\n",
    "    keras.layers.Bidirectional(\n",
    "        keras.layers.GRU(128)\n",
    "    ),\n",
    "    keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "\n",
    "history = model.fit(X_train, y_train, epochs=6,\n",
    "                    batch_size=32, validation_split=0.2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
